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Peer  Review  Process  and  Accreditation  of  Models 

Jerry  Banks 
CPT  Eric  Holmes 

Georgia  Institute  of  Technology 

This  research  prescribes  a  general  fr2unework  for  accreditation 
of  large  scale  military  simulation  models  such  as  those  used  by 
CAA.  Accrediting  a  model  for  a  specific  use  is  the  direction  toward 
which  the  Army  is  moving.  A  recent  memorandum  by  the  DUSA(OR) 
outlines  a  general  plan  for  overseeing  the  accreditation  of  models 
as  a  necessary,  time-saving  procedure.  The  memorandum  suggests  that 
verification  and  validation  (V&V)  will  be  required  for  each  model 
during  model  development  and  accreditation  will  be  reviewed  prior 
to  each  use  of  the  model.  This  research  includes  an  exeuoination  of 
peer  reviews  completed  at  CAA  for  the  purpose  of  gaining  insight 
into  the  existing  accreditation  process. 

1*  Introduction.  The  first  chapter  explains  how  the  need  for  V&V 
has  evolved  over  time  and  how  real  world  constraints,  e.g.,  time 
and  money,  have  necessitated  accreditation  of  models.  Formal 
definitions  for  verification,  validation,  accreditation,  and  peer 
review  are  given. 


2.  Previous  Peer  Reviews.  An  overview  of  reports  of  previous  CAA 


peer  reviews  is  presented.  Reconstructing  the  techniques  and  the 
details  of  the  analysis  used  in  these  peer  reviews  was  difficult. 
Even  though  much  effort  and  expertise  went  into  these  peer  reviews, 
little  information  could  be  gained  that  would  assist  in  the 
accreditation  process  because  of  the  lack  of  formal  documentation. 

3.  Literature  Review.  To  ensure  that  this  research  was  based  on 
prior  knowledge,  an  extensive  search  of  relevant  literature  was 
conducted.  Much  published  material  was  found  on  techniques  for  V&V. 
The  more  general  piiblished  works  are  reviewed  so  that  an  individual 
conducting  further  research  has  a  reference  point.  Additionally, 
Army  policy  and  regulations  on  topics  that  may  be  encountered  in 
the  process  of  accrediting  models  are  reviewed.  Specifically,  it 
is  noted  that  accreditation  is  mentioned  only  in  the  DUSA(OR)'s 
memorandum  on  the  subject. 

4.  A  .General  Framework  for  Accreditation  Procedures.  This  is  the 
main  body  of  the  research.  This  chapter  explains  how  the  need  for 
accreditation  arises  and  then  explains  a  general  process,  void  of 
internal,  administrative  details,  for  a  methodology  for 
accreditation.  First  a  single  analyst,  independent  of  the  study 
team  considering  an  application  for  the  model,  will  review  the 
accreditation  status  of  the  model.  If  the  model  needs 
accreditation,  a  full  peer  review  teiua  will  be  assigned  to  work 
with  the  analyst.  This  team  will  become  familiar  with  the  model, 
gain  insight  into  the  proposed  study,  examine  previous  studies 
where  the  model  was  used,  and  then  make  a  recommendatxon  as  to 


whether  the  model  will  be  accredited  for  the  proposed  study.  The 
accreditation  proponent  will  then  make  the  decision  to  accredit  the 
model.  The  methodology  as  outlined  in  this  chapter  provides  a  basis 
for  consistency  and  standardization. 

5.  Final  CQmmf»r|t-R.  A  review  of  the  major  results  of  this  research 
is  presented.  Additionally,  suggestions  for  further  research  are 
given. 


Chapter  1 
Introduction 


1.1.  Purpose. 

The  purpose  of  this  research  is  to  (1)  prepare  an  assessment 
of  the  effectiveness  of  peer  reviews  as  they  have  evolved  over  the 
past  few  years  for  verification  and  validation  (V&V)  of  large  scale 
military  models,  and  (2)  develop  a  framework  for  the  accreditation 
process.  This  research  is  being  conducted  for  the  United  States 
Army  Concepts  Analysis  Agency  (CAA)  in  Bethesda,  Maryland. 

1.2.  Background . 

The  military  makes  extensive  use  of  highly  complex  simulation 
models  for  a  variety  of  purposes.  Major  uses  include:^ 

~  Battle  planning 

-  Wartime  operations 

-  Weapons  procurement 

-  Force  sizing 

-  Human  resource  planning 

-  Logistics  planning 

>  National  policy  analysis 

Simulation  and  analytical  models  save  the  government  time  and 
money.  For  example,  to  decide  if  a  different  mix  of  scout  and 
attack  helicopters  might  be  more  effective  than  the  current  mix  in 
a  cavalry  squadron,  one  might  use  simulation  or  analytic  models. 


without  simulation  or  analytical  modeling,  a  test  squadron  would 
have  to  be  selected  and  taken  out  of  combat  ready  status  for  a 
lengthy  amount  of  time  to  achieve  the  seune  results.  A  benefit  of 
simulation  is  that  it  generates  information  about  future,  untried 
strategies  and  tactics,  and  about  the  effects  of  weapon  systems  not 
yet  tested.  This  document  concentrates  on  simulation  models. 

Many  simulation  models  currently  in  use  have  evolved  over 
time.  Some  of  these  models  are  composites  of  simpler  models, 
others  began  as  complex  models,  but  they  have  been  modified 
numerous  times  to  keep  pace  with  new  doctrine  and  a  growing 
technology.  These  models  have  thousands  of  lines  of  code  that  were 
produced  by  numerous  programmers  over  a  period  of  years.  In  many 
instances  documentation  is  .  lacking  and  in  some  instances 
documentation  is  nonexistent.  Some  of  the  older  models  were  written 
before  the  need  for  program  design  language  was  recognized.  This 
makes  a  review  of  the  code  more  difficult  for  verification  of  the 
model . 

Verification  is  a  process  of  determining  that  a  model 
accurately  represents  the  developer's  conceptual  description  and 
specification.  Validation  is  the  process  of  determing  that  a  model 
is  an  accurate  representation  of  the  intended  real-world  entity 
from  the  perspective  of  the  intended  use  of  the  model .  Both 
verification  and  validation  are  continual  processes.  There  is  no 
defined  point  signifying  that  a  model  is  verified  or  validated. 
The  most  difficult  aspect  of  V&V  is  the  determination  that  there 
has  been  sufficient  V&V  of  a  model  to  make  a  conclusive  statement 
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about  the  model's  credibility. 

Just  a  cursory  look  at  a  model's  code  and  comparing  it  to 
the  model's  requirements  is  only  the  beginning  of  the  verification 
process,  but  it  is  not  as  thorough  as  a  complete  walkthrough  of  the 
code.  To  accomplish  a  thorough  V&V,  more  techniques  and  statistical 
tests  can  be  completed  that  will  give  additional  confidence  in  the 
model.  Thus,  the  more  V&V  a  model  receives  the  better,  but  there 
is  a  point  of  diminishing  returns.*  Discovering  this  point  is  a 
difficult  but  important  goal,  because  an  extensive  V&V  is  very  time 
consuming  and  expensive.  In  the  absence  of  V&V,  analysts  have  no 
confidence  that  a  model  can  produce  output  from  which  reliable 
decisions  can  be  made.  With  insufficient  V&V,  analysts  may  still 
have  a  low  level  of  confidence  in  a  model. 

Confidence  in  a  model  is  why  V&V  has  become  such  an  important 
issue.  The  need  for  all  existing  models  to  be  verified  and 
validated  is  important  but  generally  considered  more  expensive  than 
if  the  models  were  verified  and  validated  as  they  were  developed.* 
At  present  CAA  has  contracted  V&V  during  the  development  of  a  new 
model.  Global  Deployment  Analysis  System  (GDAS)  which  evaluates  the 
capabilities  and  requirements  of  the  mobilization  base  and 
deployment  status  of  the  deploying  forces  and  provides  input  to  CAA 
combat  models.*  Performing  V&V  on  a  model  while  it  is  being 
developed  increases  the  model's  cost  approximately  fifteen  per 
cent.*  The  model  can  be  changed  more  easily  during  its 
developmental  stage.  To  perform  V&V  on  all  existing  models  is  a 
more  expensive  and  extensive  task. 
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As  an  alternative  to  a  full  V&V,  the  Army  is  promoting 
accreditation  of  a  model  for  a  specific  use.”  Accreditation  is  the 
process  of  certifying  that  a  computer  model  has  achieved  an 
established  standard  such  that  it  can  be  applied  for  a  specific 
purpose.  The  accreditation  procedure  recognizes  that  V&V  of  a  model 
are  continual  processes  and  that  full  validation  of  the  model  may 
not  be  technically  or  economically  feasible. 

The  standard  Army  approach  to  accredit  any  analytical  research 
is  by  peer  review,  which  is  a  detailed  examination  by  expert 
observers  who  are  independent  of  the  analysts  conducting  the  work.^ 
Likewise,  model  accreditation  may  be  accomplished  by  peer  review.” 
The  peer  review  is  also  considered  an  appropriate  mechanism  for 
obtaining  prescriptions  for  improvement.  Part  of  the  process  of 
verification  includes  adjusting  a  model  to  correct  coding  mistakes 
instead  of  just  listing  a  model's  problems. 

The  effective  use  of  peer  reviews  as  an  assessment  tool  is 
dependent  upon  the  consistency  of  evaluation  methodology  among  the 
individuals  within  a  review  group  and  between  different  review 
groups.  A  standard  methodology  for  a  peer  review  would  stabilize 
the  differences  among  peer  review  teams.  For  exeunple,  one  team 
might  be  more  thorough  than  another  team.  The  confidence  in  all 
peer  reviews  can  only  be  as  high  as  the  confidence  in  the  peer 
review  conducted  with  the  least  raount  of  effort.  In  the  long  run, 
there  will  be  a  degradation  in  the  quality  of  work  without  a 
prescribed  standard.  A  standard  to  judge  peer  reviews  will  keep  the 
level  of  effort  consistently  high.  CAA  would  benefit  from  knowing 
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that  all  pear  reviews  of  its  nodels  were  completed  with  the  seuae 
efficacy  and  consistency. 

1.3.  Current  Issues . 

Model  V&V  and  accreditation  are  processes  that  cannot  be 
immediately  accomplished  by  edict  alone.  There  are  many  questions 
that  are  unanswered,  and  new  research  is  uncovering  more  questions. 
V&V  in  many  ways  is  very  elusive.  No  one  can  predict  or  pinpoint 
the  exact  amount  of  effort  needed  to  ensure  that  a  model  is 
verified  and  validated.  Clayton  J.  Thomas  linked  V&V  to  the 
uncertainty  principle.  This  is  more  commonly  referred  to  as  the 
Hawthorne  Effect."  The  very  act  of  collecting  data  biases  the 
validation  process,  id  est,  if  a  subject  knows  that  he  is  being 
observed  he  will  perform  differently.^"  Currently,  some  of  the 
problems  plaguing  analysts  are  verifying  and  validating  several 
models  used  in  conjunction  with  each  other,  not  enough  usable 
combat  data,  and  a  world  changing  faster  than  it  takes  to  V&V  a 
model. Another  problem  that  modelers  have  encoiintered  has  been 
adjusting  input  data  so  that  the  output  madces  sense  to  such  a 
degree  that  the  input  data  becomes  unrealistic. 

In  the  eJssence  of  a  real  world  comparison,  no  matter  how 
thorough  is  the  V&V,  the  model  may  be  incorrect.  Modeling  is  very 
difficult  and  the  forces  interacting  on  events  are  numerous.  One 
recognized  problem  is  that  all  models  seem  to  simulate  combat  at 
a  faster  rate  than  live  combat.^’  Sometimes  the  mathematical 
principles  of  a  model  do  not  exactly  represent  the  real  world. 
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Outpuc  may  appear  reasonable  for  a  defined  range  of  input  values, 
but  unrealistic  outside  this  range. 

without  V&V,  a  model's  credibility  may  never  be  assured.  For 
an  exhaustive  V&V,  the  model  may  become  obsolete  before  the  V&V  is 
completed.^*  Accreditation  is  a  worthwhile  concept,  but  it  does  not 
lessen  the  need  far  continuing  V&V.^*  There  are  no  easy  answers. 
Investigation  should  continue  in  the  application  of  V&V,  because 
there  are  many  benefits  from  credible  simulation. 
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ChaptMT  2 


Previous  Peer  Reviews 


2.1.  General . 

In  the  last  several  years  CAA  has  completed  several  peer 
reviews  on  various  models.  A  listing  of  all  peer  reviews  examined 
In  this  study  Is  found  In  Appendix  A.  The  final  reports  of  these 
peer  reviews  range  from  briefing  slides,  to  office  memoranda,  to 
printed  and  bound  documents.  Some  of  these  peer  reviews  were 
actually  completed  by  peers,  others  were  not.  Most  peer  reviews  are 
Indistinguishable  as  to  whether  they  were  true  peer  reviews 
completed  by  outside  analysts  or  they  were  completed  by  model 
users . 

The  purpose  of  this  chapter  Is  to  learn  from  previous  peer 
reviews  and  to  use  this  knowledge  for  the  development  of  a 
methodology  for  accreditation  of  models.  This  chapter  Is  not  a 
critique  of  the  effort  and  expertise  Involved  In  any  of  the  peer 
reviews  completed  at  CAA.  Yet,  a  lack  of  formal  documentation  of 
these  peer  reviews  has  resulted  In  less  Insight  than  desired  for 
the  development  of  a  plan  for  accreditation  of  models.  Another 
problem  In  the  lack  of  consistent  documentation  Is  that  the  outside 
reader  can  not  distinguish  which  peer  reviews  were  thorough  and 
completed  with  extensive  effort  and  which  peer  reviews  were 
completed  with  less  extensive  effort.  The  lack  of  docximentatlon  In 
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the  reports  leads  the  outside  reader  to  a  low  level  of  confidence 
for  all  of  the  peer  reviews,  even  though  many  of  the  peer  reviews 
are  probably  completely  correct  and  accurate.  No  other  government 
agencies'  peer  review  reports  were  reviewed  and  no  comment  is  made, 
nor  should  inferences  be  made,  as  to  the  comparison  of  peer  reviews 
at  CAA  and  peer  reviews  at  any  other  agency. 

These  reports  vary  in  their  thoroughness,  because  of  the 
absence  of  consistent  definitions  for  verification,  validation, 
and  accreditation.  One  peer  review  claimed  to  validate  a  model  by 
"determining  the  model  is  not  invalid."^®  "Not  invalid"  does  not 
mean  "valid"  just  as  "not  guilty"  does  not  mean  "innocent."  This 
peer  review  can  not  be  considered  a  validation.  Some  of  the  peer 
reviews  are  verifications,  some  validations,  some  recommendations 
for  improvements,  and  some  are  just  comments.  None  were 
accreditations.  Adherence  to  strict  definitions  of  verification, 
validation,  and  accreditation  will  help  distinguish  the  reports  as 
to  their  type,  and  raise  the  reader's  confidence  level  in  these 
documents . 

2.2.  Techniques  Used  at  CAA. 

The  overview  of  all  peer  reviews  on  models  at  CAA  in  the  past 
few  years  has  revealed  the  use  of  common  techniques.  These 
techniques,  for  the  most  part,  are  not  explicitly  stated  in  the 
peer  review  reports.  The  next  three  sections  review  the  most  used 
techniques  for  verification  and  validation. 
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2.2.1.  Verification. 

Most  recent  model  verifications  have  preferred  the  use  of  one 
or  more  of  the  following  techniques: 

algorithm  critique 
expert  opinion 
review  of  documentation 
comparison  to  other  models 

Algorithm  critique  is  a  review  of  the  methodology  used  in  the 
model,  and  the  inspection  of  the  code  to  insure  that  it  reflects 
the  methodology  correctly.  Expert  opinion  is  an  inspection  of  the 
modules  to  determine  how  reality  is  represented.  Review  of 
documentation  is  a  study  of  past  V&V,  and  any  explanation  of  the 
model  including  what  it  does  and  how  it  does  it.  Comparison  to 
other  models  is  checking  an  algorithm  from  a  model  against  a 
similar  algorithm  in  a  verified  model.  Other  techniques  are 
available,  but  the  above  mentioned  techniques  were  found  to  be 
common  in  their  use. 

The  following  paragraphs  are  representative  examples  of 
information  about  a  model  that  was  gained  using  the  above  listed 
techniques  of  verification.  These  examples  demonstrate  the 
usefulness  of  verification.  These  and  many  other  examples  are 
derived  directly  from  the  peer  reviews  at  CAA. 

In  many  cases  mistakes  were  discovered  in  the  logic  of  the 
model  and  the  reports  were  useful  in  this  regard.  Additionally, 
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functional  area  experts  on  the  peer  review  teas  have  uncovered  eany 
inconsistencies  in  sonenodels  being  verified  at  CAA,  indicating  the 
usefulness  of  expert  opinions. 

Overwhelmingly,  peer  reviews  conducted  at  CAA  have  reported 
a  lack  of  documentation  on  any  past  reviews  of  the  models,  and  also 
a  lack  of  dociamentation  for  explanation  of  the  code.  In  one 
instance  the  only  knowledge  of  the  program  was  in  the  heads  of  the 
users,  and  this  was  lost  when  they  moved. Those  analysts  assigned 
to  verify  this  model  had  a  difficult  task.  Reviewing  documentation 
is  an  excellent  place  to  start  a  model  verification,  but  the 
analysts  were  handicapped  without  this  documentation  of  the  code. 

2.2.2.  Validation. 

There  have  been  only  a  few  peer  reviews  for  the  purpose  of 
validation  completed  in  the  past  few  years  at  CAA.  The  technic^ues 
employed  included: 

-comparing  the  model  results  to  combat 
-comparing  the  model  results  to  the  results 
from  training  exercises 
-comparing  the  model  results  to  the  results  of 
other  models 

-comparing  the  model  results  to  expert 
opinion. 

The  following  paragraphs  are  exaBg>les  of  validation  techniques  used 
at  CAA. 
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A  Study  completed  in  1986  validated  CFAW  concluding  that  the 
results  reasoned>ly  compared  to  combat  results  from  the  Falkland 
Islands. “  In  addition,  one  of  the  modules  was  chemged  to  reflect 
lessons  learned  from  this  study.  Using  this  technique  is  not  always 
practical  because  of  the  necessity  of  collecting  real  battle  data 
which  are  rarely  avail2d>le. 

The  Ardennes  contract  effort  built  a  data  base  from  the  Battle 
of  the  Bulge  in  order  that  a  future  study  might  validate  the 
Concepts  Evaluation  Model  (CEM) ,  then  the  Force  Evaluation  Model 
and  Joint  Theater- Level  Simulation  (JTLS) Collecting  data  based 
on  older  weapons  has  its  obvious  drawbacks,  but  this  procedure  can 
be  useful  in  model  validation  because  it  reflects  the  intangibles 
of  combat  that  are  not  so  easily  written  into  an  algorithm.  The 
major  disadvantage  is  the  lack  of  accurate  data  on  all  the  outputs 
a  model  may  generate. 

A  major  training  exercise  can  provide  much  useful  data  for 
model  validation  and  is  generally  recognized  as  the  next  best 
source  of  information  compared  to  live  combat.  The  ongoing 
exercises  at  the  National  Training  Center  (NTC)  at  Fort  Irwin, 
California  are  a  potential  source  of  data  for  model  validation,  but 
the  exercises  are  usually  at  a  smaller  scale  than  that  at  which  CAA 
models.  CAA  models  theater  level  campaigns  which  are  too  large  to 
be  conducted  at  NTC  where  battalions  are  the  usual  pairticipants . 

COMO,  a  stochastic,  critical-event-stepped,  Monte  Carlo  model, 
has  been  validated  by  comparing  results  to  exercise  Heumner  87-1.^* 
This  peer  review  was  well  documented.  The  inconsistencies  of  the 
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model  and  a  non-live  fire  air  defense  training  exercise  were 
explained  in  detail  in  the  peer  review.  The  major  lesson  learned 
from  this  peer  review  is  that  the  prescription  for  collecting 
necessary  data  must  be  well  conceived  in  advance  of  the  exercise. 

Validating  a  model  based  on  other  models  is  another  method 
used  at  CAA.  Micro-FASTALS ,  an  IBM  personal  computer  compatible 
version  of  Force  Analysis  Simulation  of  Theater  Administrative  and 
Logistic  Support  (FASTALS) ,  was  compared  to  FASTALS  for  validation. 
The  results  demonstrated  that  Micro-FASTALS  is  not  as  detailed  as 
FASTALS,  but  still  achieves  consistent  output  in  a  more 
transportable  hardware  system. 

Validating  a  model  based  on  other  models  has  its  merits,  but 
it  has  a  limiting  effect.  The  upper  bound  of  how  accurately  a  model 
represents  a  system  is  constrained  by  how  accurately  the  comparison 
model  represents  reality.  Validation  by  model  comparison  will 
always  be  limited  in  this  manner.  Currently  a  larger  problem  for 
using  model  comparison  as  a  method  of  validation  is  that  there  are 
so  few  models  that  have  been  thoroughly  and  properly  validated,  and 
existing  dociimentation  is  generally  below  an  acceptable  standard 
for  any  confidence  in  the  peer  review. 

Another  method  of  model  validation  is  that  of  comparing  model 
output  to  expert  opinion.  This  is  analogous  to  comparing  results 
of  a  model  to  live  combat  data  but  using  a  data  base  that  comes 
directly  from  the  memory  of  one  of  the  participants.  As  years  pass 
since  the  last  extensive  conflict  involving  the  United  States,  this 
reservoir  of  combat  experience  will  dwindle.  In  validating  ffASRAMP 
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it  was  determined  from  expert  opinion  that  the  artillery  loses  were 
too  low.^‘  Otherwise,  this  method  of  validation  has  not  been  used 
that  often  at  CAA. 


2.2.3.  Accreditation . 

Accreditation  is  more  economically  feasible  than  full 
validation.  Accreditation  will  not  replace  the  need  for  validation, 
but  its  intent  is  to  certify  a  model  for  a  specific  purpose  or  use. 
Even  though  technic[ues  of  accreditation  are  similar  to  validation 
techniques,  comparison  to  combat  results  is  not  necessary  for 
accreditation. 

In  the  past  two  years,  no  report  from  CAA  proclaims  itself  to 
be  an  accreditation  of  a  model,  in  fact,  no  review  actually 
attempted  to  achieve  the  results  of  accreditation  as  defined  in 
this  report.  No  peer  reviews  were  written  to  certify  a  model  for 
a  particular  use,  so  there  is  no  assessment  of  any  peer  review 
accreditation  of  models. 

2.3.  Level  of  effort. 

Some  measure  of  level  of  effort  per  amount  of  output  could  be 
very  useful .  It  would  be  convenient  to  be  ed>le  to  predict  that  a 
structured  wallcthrough  costs  a  certain  dollar  amount,  or  that  a 
comparison  to  real  battle  data  will  cost  another  amount.  Most  peer 
reviews  at  CAA  did  not  include  a  cost  of  the  report,  and  none 
included  a  price  decomposition  into  how  resources  were  expended. 
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In  addition  the  models  vary  significantly  in  size,  documentation 
of  code,  program  design  language,  complexity,  and  purposes;  so  that 
a  specific  prediction  of  cost  for  a  specific  technique  becomes 
unrealistic.  Two  studies  that  validated  two  respective  models  using 
live  combat  data  varied  greatly  in  cost.  The  Ardennes  study  had 
a  cost  of  $375,000  and  the  Falkland  study  had  a  cost  of  $40, 000. 
It  would  be  convenient  for  CAA  to  know  how  much  V&V  it  could  get 
for  a  specific  dollar  amount,  but  this  is  very  model  dependent  and 
must  be  handled  on  a  case  by  case  basis. 

2.4.  Conclusions . 

The  results  of  this  review  of  past  CAA  studies  on 
verification,  validation,  and  accreditation  are  as  follows: 

(1)  There  has  been  inconsistency  in  the 
definitions  of  verification  and  validation. 

(2)  There  is  no  standardized  methodology  for  V&V 
currently  in  use. 

(3)  There  is  incomplete  docximentation  within  the  peer 
reviews  that  have  been  conducted. 

2.5.  Recommendations . 

A  general  framework  is  needed  for  consistent  verification  and 
validation  of  models.  All  models  should  have  a  well  documented 
account  of  their  verification  and  validation.  This  document  should 
include  at  least  the  following: 
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-The  purpose  of  the  nodel 
-A  classification  of  the  model 

-A  listing  of  important  implicit  and  explicit  assumptions 
-A  range  of  values  for  significant  input  values 
-A  listing  of  verification  and  validation  techniques  used 
-All  changes  made  to  the  model 

-Identification  of  any  areas  that  might  become  a  problem 
under  certain  conditions 
-Data  on  level  of  effort. 

CAA  should  ensure  that  all  peer  reviews  are  exeunined  by  a  higher 
authority  than  those  conducting  the  peer  review.  This  may  be  in  the 
form  of  a  Product  Review  Board.  This  will  insure  a  level  of 
consistency  in  the  peer  reviews.  In  addition  a  plan  for 
accreditation  should  also  be  developed  for  application  in  those 
instances  when  a  model  needs  to  be  certified  for  a  specific 
purpose,  and  it  is  not  economically  feasible  for  a  full  validation 
to  be  conducted.  Chapter  Four  presents  such  a  plan. 
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Ch^^MT  3 


Literatiure  R«vlev 

3.1.  Recent  studies  on  V&V. 

There  have  been  several  books  and  reports  that  have  suggested 
methodologies  and  techniques  for  V&V.  All  of  these  reports 
explained  the  need  for  V&V,  and  these  are  all  in  agreement  with 
respect  to  the  reasons  for  conducting  V&V.  The  prescriptions  for 
V6V  differ  somewhat,  but  the  outline  of  some  of  these 
recommendations  for  V&V  follow. 

In  1969,  Robert  E.  Schellenberger  from  Temple  University, 
wrote  "Criteria  for  Assessing  Model  Validity  for  Managerial 
Purposes."  Even  though  this  paper  is  twenty  years  old,  it  is 
applicable  today.  This  paper  describes  measures  for  judging  %diether 
a  model  has  been  properly  validated.  Three  kinds  of  validity  are 
explained:  technical  validity,  operational  validity,  and  dynamic 
validity.  Each  of  these  kinds  of  validity  are  further  categorized. 
Additionally,  an  analysis  of  assuaqptions  is  presented.  The  paper 
is  very  thorough  in  its  explanations  of  validation  criteria.  This 
paper  was  written,  not  for  an  analyst,  but  for  a  reviewer  of  a 
model  validation  which  maJces  it  good  background  reading  for  the 
study  of  accreditation. 

The  National  Bureau  of  Standards  published  a  guidebook,  in 
1982,  Software.  Validation.  Verification,  and  Testing  Technique 
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Tool  Reference  Guide,  with  thirty  techniques  for  V&V.  Each  of  these 
techniques  is  clearly  defined,  explained  in  detail,  and  an  exanple 
of  its  use  is  given.  In  addition,  conaents  about  cost  and  effort 
are  added.  A  bibliography  is  also  included  for  each  technique  for 
the  reader's  further  reference.  Mo  fomal  proposal  is  given  in  this 
guidebook  as  to  which  techniques  are  better.  The  guidebook  is 
simply  a  reference  source. 

Another  National  Bureau  of  Standards  publication,  A  Software 
Igsting — Methodology  Using  the  Cvclomatic  Complexity  Metric, 
published  in  1982,  explains  how  to  test  a  program  during  the 
development  and  maintenance  stages  of  a  model.  This  piiblication 
specifically  is  concerned  with  verifying  that  a  model  stays  within 
reasonable  computational  time.  The  book  is  a  practical  application 
of  complexity  theory  and  maintaining  a  program  polynomial  time 
solvable.  Changes  made  to  the  model  during  accreditation  have  to 
be  checked  to  ensure  that  computational  time  does  not  grow 
exponentially.  The  techniques  presented  in  this  publication  can  be 
used  to  prevent  exponential  growth  of  computational  time. 

Wayne  P.  Hughes  edited  a  collection  of  papers  in  1984  for  the 
Military  Operations  Research  Society.  The  book.  Military  Modeling, 
has  two  chapters  -  one  by  Clayton  J.  Thomas  and  the  other  by  Wilbur 
B.  Payne,  which  e^ound  in  depth  on  the  importance  of  V&V.  The 
final  chapter  in  this  book,  written  by  Stephen  Leibholz,  is  a 
thought  provoking  coverage  of  twenty  open-ended  questions  that  a 
modeler  should  ask  about  a  model.  Many  of  the  questions  and 
subsequent  discussions  are  pertinent  to  accreditation  of  models. 
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It  is  an  excellent  reference  source  for  conducting  V&V  of  military 
models. 

In  1985,  W.  J.  Quirk  edited  a  book.  Verification  and 
Validation  of  Real-time  Software.  Several  methods  and  techniques 
for  V&V  are  discussed  but  not  in  any  detail.  This  book  does  not 
provide  much  information  that  can  not  be  learned  elsewhere 
pertaining  to  V&V.  This  book  is  another  proponent  for  V&V  during 
the  development  stage  of  a  model. 

Banks ,  et  al . ,  of  the  Georgia  Institute  of  Technology  prepared 
a  docximent  entitled  The  Verification  and  Validation  of  Simulation 
Models.  This  document,  prepared  for  CAA  in  1986,  outlines  a  process 
for  V&V  during  a  model's  development.  The  seven  steps  of  model 
development  are: 

System  feasibility 

Requirements  definition 

Preliminary  design 

Detail  design 

Coding 

Testing 

Operations  and  maintenance. 

This  document  suggests  that  a  model's  progress  should  not  proceed 
to  the  next  stage  until  the  current  process  is  verified  and 
validated.  The  research  attempts  to  quantify  how  much  V&V  is  needed 
using  a  series  of  Go/No  Go  processes  at  the  different  stages  of 
model  development.  Included  in  this  document  is  a  description  of 
how  these  procedures  should  be  applied.  The  techniques  for 
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verification  explained  in  this  report  are  documentation, 
operational  graphics,  program  design  language,  structured 
programming,  structured  walkthrough,  and  traces.  Explanations  for 
validation  techniques  are  presented  for  comparison  to  other  models, 
consistency  checks,  documentation,  event  validity,  extreme 
condition  test,  face  validity,  historical  data,  and  historical 
methods.  The  document  gives  an  appreciation  of  how  difficult  it  is 
to  quantify  the  necessary  amovmt  of  V&V  for  a  model. 

The  General  Accounting  Office  published  a  report,  POD 
Simulations;  Improved  Assessment  Procedures  Would  Increase  the 
Credibility  of  Results,  in  1987.  The  GAO  study  concerned  an 
analysis  of  the  credibility  of  three  Army  simulation  models.  This 
report  concluded  that. 

The  Department  of  Defense  should  adopt  or 
develop  and  implement  guidance  on  producing, 
validating,  documenting,  managing, 
maintaining,  using,  and  reporting  simulations 
of  weapon  system  effectiveness.^” 

This  report  also  stated  that,  "Credibility  would  be  helped  by 
better  documentation  of  the  verification."’^  Since  the  publication 
of  this  report,  more  emphasis  has  been  placed  on  insuring  that 
simulation  models  are  more  credible. 

In  1988  Banks,  et  al.,  prepared  another  document,  entitled 
Testing.  Understanding,  and  Validating  Complex  Simulation  Models. 
New  ideas  for  performing  V&V  at  CAA  are  suggested  from  analogies 
made  to  other  systems  in  which  V&V  is  prevalent.  Control  charts, 
acceptance  sampling,  fractional  factorial  design,  and  cluster 
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analysis  are  introduced  and  explained  as  statistical  nethods  that 
may  be  modified  or  used  directly  for  verification.  Unusual 
applications  of  sensitivity  analysis  and  Turing  tests  to  the 
validation  of  complex  military  simulations  are  described.  These 
techniques  were  used  to  validate  models  that  had  already  been 
implemented.  Sensitivity  analysis  and  Turing  tests  have  promise  for 
use  in  model  accreditation. 

The  Office  of  the  Deputy  Under  Secretary  of  the  Army  for 
Operations  Research  maintains  a  catalog  of  combat  simulations 
entitled  American  British  Canadian  Australian  Catalog  of  War  Games. 

Igflininq _ Games,  and  Combat  Simulations.  The  latest  version, 

released  in  1988,  lists  over  one  hundred  combat  models.  For  each 
model  listed,  a  brief  description,  usually  two  pages,  and  a  point 
of  contact  is  given.  The  eleventh  edition  of  the  Catalog  of 
Waroaming  and  Simulation  Models  was  published  in  August  1989  and 
is  now  availed)le  from  the  Defense  Technology  Information  Center. 
It  describes  models  used  by  all  U.  S.  services.  An  analyst  needing 
a  simulation  for  a  study  can  use  either  of  these  catalogs  as  a  good 
initial  source  for  an  appropriate  model. 

A  short  article,  "Verification  and  Validation:  A  TRAC 
Approach,"  by  Major  Steven  Flanagan  outlined  a  methodology  for  V&V 
that  was  used  at  TRAC  in  1988.  Flanagan's  approach  to  verification 
is: 

(1)  Methodology  proposal  review 

(2)  Preliminary  design  review 

(3)  Detailed  design  review 
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(4)  Structured  wallcthrough 

(5)  Data  review 

(6)  Check  run. 

His  approach  to  validation  includes: 

(1)  Check  run 

(2)  Interfunctional  sensitivity 

(3)  Surface  validity 

(4)  Sensitivity  analysis 

(5)  V&V  products  and  by-products. 

For  all  the  government  agencies,  in  the  Army  and  elsewhere,  there 
is  very  little  pxiblished  material  on  exactly  how  an  agency 
accomplishes  V&V.  Practical  application  of  which  techniques  of 
verification  and  validation  an  agency  uses  should  not  be  a  trade 
secret,  and  this  az±icle  simply  explains  how  one  organization 
conducted  V&V. 

3.2.  Department  of  the  Army  policy. 

Currently,  there  are  few  specific  regulations  pertaining  to 
V&V  or  accreditation.  Generally  the  practice  has  been  for  local 
commands  to  develop  policy,  with  little  Department  of  the  Army 
directed  regulation.  The  following  overview  of  Army  publications 
may  not  be  exhaustive,  but  the  overview  covers  the  more  important 
aspects  that  concern  CAA.  This  overview  is  presented  as  an  initial 
guide  to  Army  publications  concerning  model  accreditation. 

The  Deputy  Under  Secretary  of  the  Army  for  Operations  Research 
[DUSA(OR) ]  has  issued  a  memorandum,  "Verification  and  Validation 
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and  Accreditation  of  Models,"  that  has  a  specific  plan  for  V&V  of 
models  and  for  model  accreditation.  This  memorandum  is  applicedsle 
to  CAA.  A  copy  of  this  memorandum  is  included  as  Appendix  B  of 
this  report. 

The  regulation  for  CAA,  United  States  Army  Concepts  Analysis 
Agency .  (AR  lO-SS)  does  not  specifically  mention  V&V  of  combat 
models.  This  is  a  short  regulation  describing  duties,  functions, 
and  responsibilities  of  CAA.  Accreditation  and  V&V  of  models  are 
internal  requirements  that  allow  CAA  to  effectively  accomplish  its 
mission  as  defined  in  this  regulation. 

The  Army  Model  Improvement  Program  is  an  Army  regulation  (AR 
5“11)  that,  by  its  purpose,  seems  to  govern  V&V.  This  regulation 
is  in  the  process  of  being  re-written.  Many  changes  are  expected, 
and  it  will  be  directive  in  nature  in  that  it  will  require 
validation  and  accreditation  of  all  models  to  be  used  in  formal 
Army  studies  and  acquisition  decisions.  It  will  allow  directors  of 
analytical  agencies  to  use  discretion  in  the  choice  of  methods  for 
V&V  and  accreditation. 

Army  Studies  Analysis  (AR  5-5)  describes  how  studies  are  to 
be  written.  As  to  whether  V&V  or  accreditation  of  models  are 
defined  to  be  studies  that  must  follow  this  regulation  is  not 
clearly  Indicated.  If  the  V&V  is  accomplished  during  the 
development  of  a  model  then  this  study  would  be  covered  by  this 
regulation.**  Even  if  not  required  for  accreditation,  this 
regulation  and  the  accompanying  pamphlet  outline  a  method  for 
effectively  conducting  a  study.  The  pamphlet  lists  six  phases  of 
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a  study:  (1)  initiation,  (2)  validation,  (3)  development  and 
conduct,  (4)  evaluation,  (5)  application,  and  (6)  documentation  and 
reporting.  The  pamphlet  also  lists  duties  and  responsibilities  for 
the  members  of  a  study  teeun.  An  analyst  can  not  go  wrong  by 
following  the  procedures  in  this  pamphlet. 

Threat  Support  to  U.  S.  Army  Force  Combat  and  Material 
Development  (AR  381-11)  is  one  of  the  few  Army  regulations  that 
specifically  concerns  the  topic  of  VfiV.  Any  result  from  a  model 
simulation,  under  the  jurisdiction  of  the  Army  Model  Improvement 
Program  (AMIP) ,  which  affects  the  threat  must  be  sent  to  the  Office 
of  Deputy  Chief  of  Staff  for  Intelligence  (ODCSINT)  where  the 
threat  results  are  validated.  This  regulation  specifically  mentions 
that  CAA  will  obtain  threat  data  for  input  from  ODCSINT.*"  Having 
a  specific  outside  agency  validate  the  threat  results  from  a  model 
reduces  CAA's  need  for  an  intelligence  expert  on  a  peer  review 
team. 

A  regulation  that  does  not  seem  to  directly  pertain  to  V&V  is 
Configuration  Management  (AR  70-37) .  This  regulation  covers 
management  of  any  item  that  has  been  designated  by  the  government 
as  a  configuration  item.  If  a  model  is  so  designated,  then  this 
regulation  would  have  to  be  followed  to  affect  changes  to  the  model 
as  in  V&V  and  accreditation. 

There  are  other  regulations  that  may  become  involved  in  the 
process  of  model  accreditation,  but  they  will  be  very  specific  with 
respect  to  the  topic  covered  The  reference  list  in  the  Army  Model 
Improvement  Program  is  thorough  enough  to  begin  a  regulation  review 
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that  night  b«  necessary  for  a  peer  review. 
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Chaptar  4 


A  General  Fraaework  for  Accreditation  Procedures 

4.1.  General . 

This  chapter  presents  a  general  framework  for  an  accreditation 
methodology.  Accreditation  of  a  model  is  a  cost  effective 
alternative  to  V&V.  The  process  of  V&V  can  be  streeunlined  so  that 
the  model  can  only  be  accredited  for  a  specific  purpose,  and  not 
for  all  possible  applications. 

Ideally,  every  model  should  be  verified  and  validated  during 
model  development.  Understanding  that  not  all  models  in  the 
inventory  were  verified  and  validated  during  their  development,  it 
may  become  necessary  to  accept  the  fact  that  a  model  may  never  be 
verified  and  validated.  A  thorough  process  could  take  so  much  time 
that  the  study  results  may  become  obsolete  or  the  decision  making 
may  be  made  before  the  model  is  verified  and  validated. 
Realistically,  accreditation  or  a  review  of  accreditation  of  a 
model  prior  to  each  use  may  be  the  only  acceptable  practice  for 
models  that  were  never  verified  and  validated  during  their 
development. 

A  model  may  be  in  use  for  a  long  enough  time  that  tactical  emd 
technical  changes  may  have  occurred  in  the  real  world.  There  may 
be  a  modification  of  input  data,  or  there  may  have  been  an 
additional  module  added  to  the  model.  These  are  all  reasons  why  a 
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model,  accredited  for  a  specific  purpose,  is  not  guaranteed  to 
remain  credible. The  model  users  should  be  aware  of  this 
possibility  and  be  prepared  to  report  discrepancies  of  a  model  when 
these  discrepancies  are  noticed. 

The  general  idea  is  to  insure  that  all  studies  using 
simulation  will  be  conducted  using  accredited  models.  A  plan  must 
be  established  such  that  a  simulation  model  is  accredited  by  a  peer 
review  te2uii  that  is  independent  of  the  model  developers  and  model 
users . 

If  time  or  resources  preclude  an  independent  teeun  model 
accreditation,  or  if  this  procedure  is  not  followed;  the  final 
report  should  reflect  whatever  deviations  have  occurred.  Future 
users  of  the  model  should  be  informed  of  the  limitations  of  a 
shortened  or  non-independent  accreditation  process. 

4.2.  The  need  for  accreditation. 

There  are  several  events  that  will  cause  CAA  to  initiate  the 
accreditation  process.  First,  CAA  must  initially  accredit  all 
models  for  which  it  is  the  accreditation  proponent.  Thereafter,  CAA 
may  be  tasked  to  accredit  these  models  whenever  the  Army  Models 
Committee  deems  necessary.  CAA  may  also  be  tasked  to  accredit  a 
model  that  a  different  agency  has  decided  to  use  in  a  study.  CAA's 
performing  the  accreditation  in  this  instance  insures  the 
independence  of  the  peer  review.  Internally,  CAA  may  wish  to 
accredit  a  model  for  an  in-house  study.  The  independence  of  the 
peer  review  team  performing  the  accreditation  is  the  responsibility 
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of  CAA  in  this  instance. 

Another  reason  for  initiating  an  accreditation  process  could 
be  that  the  model  may  not  be  performing  as  it  was  designed.  At  the 
level  above  the  user  of  the  model,  a  tasking  is  made  for  an  analyst 
to  accredit  the  model  in  question.  By  whatever  route  a  model  is 
introduced  to  the  accreditation  process,  the  peer  review  process 
of  accreditation  should  follow  the  steps  explained  below. 

4.3.  Review  of  Accreditation. 

Before  CAA  assigns  a  peer  review  team  to  accredit  a  model,  a 
single  analyst  should  be  assigned  to  review  the  current 
accreditation  status  of  the  model.  It  might  be  the  case  that  a 
model  has  recently  been  accredited  for  a  similar  study,  and  a  full 
accreditation  procedure  is  not  necessary.  The  individual  analyst 
should  make  a  recommendation  for  a  full  peer  review  accreditation 
of  a  model  if  he  or  she  deems  that  it  is  necessary  after 
accomplishing  the  following  tasks: 

1)  Review  all  documentation  of  any  past  V&V's  or 
accreditations 

2)  Review  the  audit  trail  of  the  model's  evolution 

3)  Know  the  effectiveness  of  each  study  completed  using 
this  model 

The  analyst  may  make  the  decision  that  the  model  is  already 
accredited  for  use.  The  decision  should  be  made  only  if  a  model  has 
been  accredited  for  a  similar  study  with  the  same  range  of  values 
for  input  data.  In  addition,  an  inordinate  amount  of  time  should 
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not  have  elapsed  since  the  nodel  was  last  accredited  for  a  similar 
study.  If  the  model  was  being  used  in  a  study  when  a  problem  had 
been  noticed  with  the  model,  the  analyst  should  recommend  a  full 
peer  review  for  accreditation  at  this  point.  Accreditation  of  a 
model  by  a  full  peer  review  is  not  necessary  each  time  a  model  is 
used,  but  the  assurance  that  the  accreditation  of  a  model  is  still 
current  is  important. 

4.4.  A  full  peer  review. 

After  the  analyst  has  made  the  determination  that  the  model 
needs  to  be  considered  for  accreditation  for  a  specific  purpose, 
a  peer  review  will  be  requested.  A  peer  review  te2uii  will  be 
assigned  to  work  with  the  original  analyst.  A  peer  review  team  of 
three  or  four  members  will  probably  be  sufficient.  The  senior 
analyst  will  become  the  team  leader.  The  peer  review  team  must  be 
aware  of  all  the  factors  that  will  influence  it.  The  amount  of  time 
and  money  allocated  for  the  accreditation  process  may  be  limiting 
factors  with  respect  to  the  depth  of  the  peer  review.  Additionally, 
Army  regulations  may  dictate  certain  procedures.  The  output  of  the 
peer  review  team  should  be  a  completely  dociunented  report.  Now  the 
peer  review  team  is  prepared  to  initiate  the  following  procedures. 

4.4.1.  Know  the  model. 

The  original  analyst  will  give  a  detailed  briefing  to  the 
other  members  of  the  peer  review  team  on  what  was  learned 
previously  about  the  model  being  accredited.  The  peer  review  team 
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will  then  become  more  familiar  with  the  model.  Some  tasks  the  peer 
review  team  should  accoa^llsh  are  as  follows: 

1)  Know  the  conceptual  model. 

2)  Know  the  iinder lying  mathematical  principles  of  the 
model. 

3)  Know  all  assumptions  of  the  model. 

4)  Know  the  type  of  model,  id  est,  deterministic  or 
stochastic. 

5)  Know  if  the  model  was  designed  for  combat  development 
or  for  training. 

4.4.2.  Know  the  study. 

The  peer  review  team  should  become  familiar  with  the  demands 
and  constraints  of  the  proposed  study.  Specifically  the  peer  review 
team  should  know  the  following: 

-The  purpose  of  the  study 

-The  objectives  of  the  study 

-The  importance  of  the  study's  results 

-The  urgency  of  the  study 

-The  range  of  the  study 

-The  type  of  study,  whether  a  training  study  or  a  combat 
development  study. 

In  addition,  information  about  the  input  data  will  be  useful  in 
gaining  a  thorough  understanding  of  the  scope  of  the  study.  The 
source  of  the  input  data,  and  whether  the  study  is  using  current 
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year  data  for  input  or  projected  data  for  input  are  two  iteas  that 
the  peer  review  team  will  need  to  consider. 

4.4.3.  Know  which  functional  areas  of  the  model  are  significant. 

A  model  might  be  more  complex  than  what  is  needed  for  a  study. 

The  peer  review  team  decides  which  functional  areas  of  the  model 
are  significant  to  the  study.  Functional  areas  of  the  model  that 
are  not  necessary  for  the  proposed  study  do  not  have  to  be 
accredited.  For  example,  if  the  study  assumes  no  chemical  agent 
attack,  then  a  module  that  models  the  degradation  of  soldiers  while 
wearing  chemical  protection  suits  does  not  need  to  be  accredited. 

4.4.4.  Know  how  the  model  was  used  previously. 

How  well  a  model  performed  in  the  past  may  give  an  indication 
as  to  how  well  the  model  may  perform  in  the  future.  Previous  users 
of  a  model  will  have  experience  and  additional  Insight  into  the 
model  which  may  be  beneficial.  The  model  may  have  been  used  in  the 
past  for  a  similar  study  and  performed  poorly.  Members  of  the  peer 
review  teeuD  should  contact  past  users  of  the  model.  The  team  should 
not  limit  its  contacts  to  the  most  recent  users  of  the  model,  but 
should  contact  a  variety  of  model  users  if  this  model  was  used  for 
a  breadth  of  applications  and  various  studies.  The  peer  review  team 
will  be  looking  for  valid  criticisms  from  former  users  of  the 
model.  This  feedback  is  worth  the  extra  effort  required  to  locate 
model  users  even  if  they  have  had  a  change  of  assignment  since  they 
used  the  model. 
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4.4.5.  Know  current  status  of  the  systee. 

The  peer  review  team  should  review  changes  in  technology  and 
tactics  that  have  occurred  since  the  model's  most  recent 
accreditation,  if  one  has  occurred.  A  thorough  research  of  all 
significant  functional  areas  is  necessary  to  ensure  that  all 
changes  are  being  considered.  In  addition,  information  on  threat 
changes  in  these  significant  fxmctional  areas  will  be  requested 
from  ODCSINT.  If  a  functional  area  is  significant  for  the  current 
study,  but  was  not  significant  the  last  time  the  model  was 
accredited,  then  significant  fimctional  areas  should  be  researched 
for  changes  since  the  verification  and  validation  of  the  model,  and 
not  just  since  the  last  accreditation. 

4.4.6.  Accredit  the  model. 

After  the  peer  review  team  has  become  familiar  with  the  model, 
the  study,  and  the  latest  updates  in  tactics  and  technology,  the 
team  must  decide  if  the  model  can  be  used  for  the  proposed  study. 
Some  additional  items  the  peer  review  team  may  wish  to  consider  are 
explained  below. 

4. 4. 6.1.  Review  the  model's  assumptions. 

The  peer  review  team  should  look  at  the  assumptions  of  the 
model.  It  must  be  determined  whether  or  not  the  study  is  going  to 
violate  any  of  these  assumptions.  A  trivial  example  is  that  a  model 
may  asstuie  a  constant  defense  budget  for  all  future  years,  but  the 
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study  might  need  to  model  the  effects  of  a  fluctuating  annual 
budget. 

4. 4. 6. 2.  Examine  the  range  of  values  for  model  input. 

The  peer  review  team  needs  to  examine  the  range  of  values  for 
significant  input  data.  This  range  must  not  violate  any  of  the 
model's  constraints  on  the  significant  variables.  A  movement  plan 
to  withdraw  all  troops  from  Germany  must  consider  all  means  of 
available  transportation.  If  the  movement  plan  involves  airlifting 
troops  back  to  the  United  States,  then  the  model's  input  -the 
number  of  aircraft  needed-  is  constrained  by  current  or  projected 
inventory. 

4.4.7.  Write  a  formal  report. 

After  all  of  the  above  items  have  been  considered,  a  formal 
report  must  be  written.  This  report  should  be  thoroughly 
documented.  Briefing  slides  may  be  included,  but  should  not  be  the 
only  written  work  produced.  This  report  should  include:  a  list  of 
what  was  reviewed,  a  full  explanation  of  the  proposed  study,  a 
listing  of  all  implicit  and  explicit  assumptions  of  the  model,  a 
recommendation  on  accreditation,  a  list  of  which  specific 
algorithms  were  considered,  and  a  list  of  any  recommendations  for 
updating  the  model.  The  peer  review  team  should  not  update  or 
change  the  model.  The  report  will  be  submitted  to  the  Army  Models 
Committee  for  accreditation  approval. 
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4.5.  CAA  accreditation  criteria. 

CAA  may  wish  to  accredit  internal  models  for  certain  studies 
even  though  accreditation  is  not  required  by  the  Army  Models 
Committee.  In  this  instance,  CAA  needs  to  establish  accreditation 
criteria.  CAA  should  task  a  senior  analyst,  not  on  the  peer  review 
team  or  on  the  study  team,  to  review  the  accreditation  report.  This 
reviewer  will  approve  or  disapprove  the  accreditation  of  the  model 
for  the  proposed  study  based  on  the  peer  review  team's 
recommendation.  If  the  peer  review  teeun  made  recommendatic.is  for 
a  change  of  the  model,  the  reviewer  will  recommend  approval  or 
disapproval  of  these  changes.  The  Director  of  CAA  will  medce  the 
decision  whether  to  implement  these  changes. 

The  reviewer  of  the  proposed  accreditation  will  base  this 
decision  on  whether  to  approve  the  peer  review  team's 
recommendation  for  accreditation  on  the  following  criteria: 

-The  model  has  been  verified  and  validated,  or 
-There  is  an  acceptance  by  model  users 
-Study  parameters  do  not  violate  model  constraints 
-Threat  data  has  been  validated  by  ODCSINT 
-All  implicit  and  explicit  assumptions  of  the  model  have 
been  identified 

-All  proposed  changes  to  the  model  have  been  checked  to 
ensure  that  these  changes  will  not  adversely  affect 
other  portions  of  the  model 

-All  other  approval  standards  of  the  Army  Model  Committee 
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have  been  met. 

In  addition,  the  formal  written  report  of  the  accreditation  should 
allow  future  analysts  to  gain  the  information  that  is  explained  in 
Section  4.3.^^ 
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Chiq^tcr  5 


Final  Comments 

This  doctiment  has  examined  docijmentation  of  results  of 
unclassified  V&V  peer  reviews  at  CAA.  From  this  examination,  a  more 
formal  fr€unework  for  conducting  peer  reviews  is  recommended. 
Documentation  of  past  V&V  efforts  has  been  inconsistent,  and  in 
many  instances  has  lacked  important  information  for  a  clear  audit 
trail.  Therefore,  an  assessment  of  the  effectiveness  of  past  peer 
reviews  in  terms  of  resources  expended  and  time  required  is  not 
possible  at  this  time.  Further  research  can  be  conducted  comparing 
cost  savings  of  accrediting  models  versus  the  loss  of  credibility 
from  not  conducting  a  full  V&V  each  time  the  model  is  used. 

A  general  outline  of  accreditation  procedures  is  proposed  in 
this  document.  Since  certain  types  of  applications  of  models  are 
required  to  be  accredited,’’  it  is  recommended  that  CAA  adopt  this 
procedure  for  model  accreditation.  Since  the  idea  of  accreditation 
of  models  is  relatively  new  and  not  a  standard  practice,  continued 
monitoring  of  this  procedure  is  recommended.  Through  use,  revisions 
may  be  made  to  enhance  this  accreditation  methodology.  More 
criteria  for  accreditation  might  be  deemed  useful  from  the 
continued  use  of  the  accreditation  process.  In  addition,  other 
steps  of  the  accreditation  process  might  becoBW  more  streamlined 
through  use  and  refinement.  The  documentation  of  future 
accreditations  of  models  will  become  more  detailed  as  time  passes. 
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and  the  accreditation  process  will  be  less  tise  consuming.  Thus, 
ongoing  review  of  this  process  is  recomnended.  Only  through 
continuous  refinement  will  this  process  of  accrediting  models 
become  viable. 
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Verification  and  Validation  (V&V)  and 
Accreditation  of  Models 


SAUS-OR 


DEPARTMENT  OF  THE  ARMY 

Office  OF  THC  UMOCW  •CCWCTABV 

WASHINGTON,  o.c.  aosiO'Oloa 

3  0  OCT  1969 


MEMORANDUM  FOR:  SEE  DISTRIBUTION 

SUBJECT:  Verification  and  Validation  (V&V)  and  Accreditation 

of  Models 


1.  PURPOSE.  This  memorandum  states  policy  and  guidance  providing 
quality  assurance  for,  and  establishing  the  credibility  of, 
models.  Included  are  all  models  used  for  studies  and  analyses, 
training,  combat  developments,  education,  operational  planning, 
testing,  and  command  decision  aids.  Immediate  actions  required  by 
this  memorandum  are  identified  in  paragraph  6c (1) .  The  VSV  and 
accreditation  policies  and  responsibilities  in  this  memorandum 
will  be  incorporated  in  the  next  revision  of  AR  5-11  (Army  Model 
Improvement  Program)  (reference  1) .  A  separate  policy  memorandum 
dealing  with  configuration  management  for  models  will  be  prepared 
and  distributed  in  the  near  future. 

2.  REFERENCES.  See  Enclosure  1. 

3.  DEFINITIONS.  See  Enclosure  2. 

4.  BACKGROUND  AND  RELATED  ACTIVITIES. 

a.  Expansion  of  the  use  of  models  by  the  Services,  Unified  and 
Specified  Commands,  Joint  Staff,  and  other  Department  of  Defense 
(DoD)  agencies  has  led  to  the  recognition  of  the  need  for  clearer 
policy  for  the  integrated  development,  acquisition,  and 
application  of  models.  The  Defense  Science  Board,  Army  Science 
Board,  Army  Audit  Agency,  and  General  Accounting  Office  have  noted 
such  needs  (see  references  2-7) . 

b.  The  Organization  of  the  Joint  Chiefs  of  Staff  (OJCS)  has 
created  the  Joint  Models  and  War  Games  Executive  Council  (JMWEC) 
(see  reference  8) .  The  purpose  of  JMWEC  is  to  improve  joint 
models  and  war  games  used  by  the  Services,  Joint  Staff,  Unified 
and  Specified  Commands,  and  other  DoD  agencies,  by  promoting 
efficiencies,  economies,  and  interoperability  in  such  models  and 
war  games.  JMWEC  is  a  forum  in  which  the  Army  can  recommend 
improved  representations  of  ground  warfare  systems,  operations, 
and  concepts  in  models  used  throughout  the  DoD  community. 
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c.  The  Army  recognizes  the  need  for  credibility  assessment  for 
Army  and  joint  models  to  provide  for  increased  acceptability. 
Reference  9  suggested  that  the  Army  Model  Improvement  Program 
(AMIP)  Management  Office,  now  part  of  the  U.  S.  Army  Model 
Improvement  and  Study  Management  Agency  (MISMA) ,  develop  a  master 
plan  and  oversee  an  overall  program  of  credibility  assessment. 

d.  The  increased  use  of  models  in  support  of  operational  test 
and  evaluation  (OT&E)  has  led  the  Director,  Operational  Test  and 
Evaluation  (DOT&E)  to  issue  policy  guidance  (reference  10)  on  V&V 
for  models  used  for  OT&E. 

e.  Under  AMIP  (reference  1),  the  Army  is  developing  and 
applying  a  family  of  analytical  models.  This  family  includes 
combined  arms  models  and  functional  area  models.  The  US  Army 
Concepts  Analysis  Agency  (CAA)  is  the  proponent  for  the  theater 
level,  combined  arms  component;  and  the  U.  S.  Army  Training  and 
Doctrine  Command  (TRADOC)  is  the  proponent  for  the  corps/division 
and  battalion  task  force,  combine  arms  components.  Item  system 
performance  data  used  in  the  models  are  provided  by  the  U.  S.  Army 
Materiel  Command  (AMC) . 

f.  The  Army  is  developing  and  fielding  a  family  of  training 
simulations  (FAMSIM) .  These  are  administered  under  AR  350-38 
(Training  Device  Policies  and  Management)  (reference  11),  which  is 
under  revision.  The  Army  Staff  proponent  for  FAMSIM  and  AR  350-38 
is  the  Director  of  Training,  Office  of  the  Deputy  Chief  of  Staff 
for  Operations  and  Plans.  For  AMC,  Program  Manager,  Training 
Devices  is  the  materiel  development  proponent  for  FAMSIM.  For 
TRADOC,  Commander,  Combined  Arms  Training  Activity  and  National 
Simulation  Center  is  the  training  development  proponent  for 
FAMSIM;  Commander,  TRADOC  Analysis  Command  (TRAC)  is  the  V&V 
proponent  for  FAMSIM;  and  Commander,  Army  Training  Support  Center 
is  the  proponent  for  AR  350-38. 

g.  Current  DoD  policy  on  V&V  is  stated  in  a  number  of 
publications,  among  which  are  references  12-14.  Applicable  Army 
policy  and  guidance  are  stated  in  references  15-20.  Reference  18 
contains  useful  guidance  for  V&V  in  general,  V&V  testing, 
independent  V&V,  and  configuration  control;  reference  19  contains 
useful  guidance  on  configuration  control,  V&V,  and  documentation; 
and  reference  20  deals  with  validation  of  threat. 

5.  POLICY. 

a.  The  Department  of  the  Army  recognizes  the  value  of  models 
to  support  training,  combat  developments,  education,  operational 
planning,  and  testing;  to  assist  decision  making  through  studies 
and  analyses;  and  as  potential  command  decision  aids. 
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b.  Department  of  the  Army  policy  is  to  use  models  that  are 
widely  accepted  as  high  quality  within  the  Army  and  the  larger  DoD 
community.  This  goal  will  be  attained  through  continuous 
attention  to  quality  and  V&V,  and  continuous  commitment  to 
improving  the  modeling  process. 

c.  The  Army  has  developed  a  body  of  knowledge  of  battlefield 
phenomena  that  will  be  applied  in  representing  such  activities. 

To  the  extent  possible#  models  treating  the  same  function  will  use 
consistent  representations,  with  levels  of  detail  dependent  upon 
intended  use. 

d.  The  Department  of  the  Army  will  perform  and  document  V&V  on 
any  model  over  which  it  has  control.  V4V  is  an  Army  command 
responsibility.  V&V  of  a  particular  model  includes  preparation  of 
a  written  assessment  plan  and  a  written  report.  V&V  are 
continual,  in  the  sense  that  V&V  activities  are  repeated  as  a 
model  is  modified;  or  new  test,  exercise,  training,  or  operations 
data  become  available. 

e.  For  a  model  developed  and  used  by  a  Federally  Funded 
Research  and  Development  Center  (FFRDC)  or  contracting  firm  in 
support  of  a  major  Army  study,  the  Army  sponsor  will  ensure  that 
V&V  has  been  performed  on  the  model  and  ascertain  its  V&V  status. 
This  can  be  accomplished  by  including  a  requirement  for  V&V 
documentation,  and  stipulating  acceptability  criteria,  in  the 
Request  for  Proposal  (RFP)  and  statement  of  work  (SOW)  (or  other 
appropriate  legal  documents) . 

f.  Accreditation  is  necessary  if  a  model  is  not  fully 
validated.  The  Army  will  establish  and  update  the  accreditation 
status  of  each  standing  model  'that  supports  key  Army  processes, 
especially  a  model  that  has  repeated  use,  is  widely  distributed, 
and  supports  training  or  operations  planning.  This  requirement 
applies  whether  the  model  is  used  in-house  by  the  Army,  or 
externally  by  a  FFRDC,  a  contracting  firm,  the  Joint  Staff,  or  a 
major  combatant  commander.  The  need  for  accreditation  is  based  on 
the  recognition  that  V&V  are  continual  processes  and  that  full 
validation  of  a  model  may  not  be  technically  or  economically 
feasible.  A  model  will  be  accredited  for  a  particular  type(s)  of 
application (s) .  Regarding  frequency,  a  model  will  be  subject  to 
accreditation  when  the  model  is  proposed  for  use  for  a  new  type  of 
application,  when  a  new  reference  version  of  the  model  is 
released,  or  when  a  sufficient  period  has  passed  to  necessitate 
accreditation  activities  to  be  performed  again. 

g.  V&V  and  accreditation  will  be  included  in  an  overall 
program  of  configuration  management. 
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h.  A  standing  model  that  satisfies  one  (or  more)  of  the 
criteria  listed  below  must  be  accredited  for  that  type(s)  of 
application (s) .  Examples  are  listed  in  Enclosure  3. 

(1)  Model  that  is  used  as  a  primary  analysis  tool  in  an 
Army  cost  and  operational  effectiveness  analysis  (COEA)  for  a 
Defense  Acquisition  Board  (DAB) -level  or  Army  Designated 
Acquisition  System  (ADAP) -level  system. 

(2)  Model  that  is  used  as  a  primary  analysis  tool  in  a 
major  Army  Planning,  Programming,  Budgeting,  and  Execution  System 
(PPBES)  study;  e.g.,  Mid-Range  Force  Study  (MRFS) ,  Support  Force 
Requirements  Analysis  (SRA)  (supporting  Total  Army  Analysis) , 
Program  Force  Capability  Assessment  (PFCA) ,  OMNIBUS,  or  Total 
Logistics  Readiness  /  Sustainability  (TLR/S) . 

(3)  FFRDC-  or  contractor-developed  model  that  is  a  primary 
analysis  tool  in  a  major  study  sponsored  by  the  Army. 

(4)  Training  simulation  in  the  Army  FAMSIM. 

(5)  Joint  model  that  represents  Army  activities  and  is 
used  in  a  study  sponsored  by  OJCS  or  the  Office  of  the  Secretary 
of  Defense  (OSD) ,  or  used  by  a  combatant  commander  for  planning  or 
training.  (Generally,  Army  V4V  and  accreditation  for  a  joint 
model  will  be  limited  to  examination  of  the  representation  of  Army 
activities . ) 

(6)  Model  that  is  used  as  a  primary  analysis  tool  in  the 
planning  phase  or  evaluation  phase  of  testing  of  a  DAB-level  or 
ADAP-level  Army  system. 

* 

(7)  Decision  support  system  that  supports  analysis 
performed  by  or  presented  directly  to  Headquarters,  Department  of 
the  Army  (HQDA) . 

(8)  Model  that  is  a  primary  education  tool  in  the 
curriculum  of  the  U...  S.  Army  War  College,  U.  S.  Army  Command  and 
General  Staff  College,  or  U.  S.  Army  Logistics  Management  College. 

i .  A  model  in  one  or  more  of  the  categories  of  paragraph  5h 
and  used  by  or  for  the  Army  for  the  associated  type(s)  of 
application (s)  must  be  appropriately  accredited.  Use  of  a  model 
that  has  been  judged  non-accredited  (i.e.,  unacceptable)  for  a 
particular  type  of  application  requires  prior  approval  of  the 
chairperson  of  the  Army  Models  Committee  (see  paragraph  7c) . 
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j.  Validation  criteria  (i.e.,  standards  of  adequacy)  will  be 
identified  as  early  in  the  development  of  a  model  as  feasible. 

This  is  especially  important  for  a  model  developed  on  contract. 

k.  V4V  and  accreditation  activities  will  Include  assessments 
of  the  representations  of  concepts,  tactics,  and  doctrine — from 
both  Red  and  Blue  perspectives. 

l.  V&V  of  threat  portrayal  will  be  performed  by  the 
appropriate  intelligence  authority,  who  will  ensure  that  the 
representations  of  concepts,  tactics,  and  doctrine  are  consistent 
with  established  intelligence  positions  and  assessments. 

m.  As  relevant  test,  exercise,  training,  or  operations  data 
become  available,  they  will  be  used  to  support  validation  and 
accreditation  of  a  model.  Data  from  tests  and  exercises  are  most 
useful  for  "calibrating"  the  given  model;  i.e.,  making  adjustments 
to  input  data  or  model  logic  to  obtain  closer  agreement  with  an 
external  index. 

n.  For  a  model  used  in  testing  (paragraph  5h(6)),  the 
accreditation  proponent  will  coordinate  with  the  appropriate  test 
and  evaluation  agency  (e.g..  Operational  Test  and  Evaluation 
Agency  (OTEA) ,  or  U.  S.  Army  Materiel  Systems  Analysis  Activity 
(AMSAA) )  regarding  acceptability  criteria. 

o.  Results  of  validation  activities  for  a  high  resolution 
model  of  a  particular  process  will  be  used  to  support  validation 
activities  of  a  more  aggregate  model  representing  the  same 
process . 

p.  A  brief  summary  of  th^  accreditation  status  of  a  model  (in 
one  or  more  of  the  categories  of  paragraph  5h)  used  or  proposed 
for  use  will  be  included  in  an  Army  Independent  Evaluation  Plan 
(lEP),  Test  Design  Plan  (TDP),  Test  and  Evaluation  Plan,  Test  and 
Evaluation  Master  Plan  (TEMP),  final  study  briefing,  or  final 
written  study  report . 

q.  Resources  required  to  perform  V&V  and  accreditation  of 
models  will  be  identified  in  Command  Operating  Budget  (COB) 
submissions . 

r.  Army  goals,  objectives,  and  plans  for  model  V&V  and 
accreditation  will  be  disseminated  throughout  the  DoD  community. 
Inconsistent  information  provided  to  organizations  external  to  the 
Army  will  undermine  improvement  efforts.  Comments  or  problems 
concerning  credibility  of  an  Army  model  will  be  brought  to  the 
attention  of  the  model's  "accreditation  proponent"  (as  defined  in 
Enclosure  2)  for  resolution. 
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6 .  PROCEDURES . 

a.  V4V.  Useful  V&V  procedures  are  shovm  in  Enclosure  4.  For 
a  number  of  reasons — variety  of  model  applications,  diversity  of 
levels  of  detail,  and  limitations  on  time  and  resources — selection 
of  V&V  procedures  and  associated  acceptability  criteria  for  a 
particular  model  will  often  require  tailoring.  Procedures  are 
suggested  for  each  of  the  life  cycle  stages  of  model  development. 
For  a  model  that  has  reached  the  application  and  post-deployment 
software  support  (PDSS)  stage  and  for  which  V&V  activities  have 
been  performed  to  only  a  limited  extent  (or  not  at  all) ,  many  of 
the  procedures  listed  under  earlier  life  cycle  stages  may  have  to 
be  applied. 

b.  Accreditation.  Useful  procedures  for  performing 
accreditation  are  listed  below.  Again,  selection  of  the 
procedures  and  associated  acceptability  criteria  will  often 
require  tailoring.  In  addition  to  a  decision  regarding 
acceptability,  accreditation  activities  may  also  provide 
recommendations  for  enhancements  to  the  model;  this  is  especially 
appropriate  if  the  model  is  judged  non-accredited  (i.e., 
unacceptable) . 

(1)  Review  of  model  documentation. 

(2)  Review  of  V&V  documentation  (plan  and  report) . 

(3)  Review  of  how  input  data  and  scenario  data  are  used  or 
modified  internal  to  the  model. 

(4)  Review  of  configuration  control  and  enhancement 
procedures . 

(5)  Recognition  of  the  extent  to  which  major  components  of 
the  model  have  been  validated. 

(6)  Recognition  of  previous  successful  applications  of  the 
model  for  similar  purposes. 

(7)  Recognition  of  acceptance  by  users. 

c.  Initial  accreditation. 

(1)  Current  model.  To  permit  initial  accreditation  of  all 
models  requiring  accreditation  (i.e.,  models  in  one  or  more  of  the 
categories  of  paragraph  5h) ,  an  initial  time-phased  program  of 
accreditation  will  be  conducted  by  appropriate  accreditation 
proponents  (Enclosure  2,  paragraph  11) .  Within  three  months  of 
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publication  of  this  memorandum,  each  Army  agency  that  is  the 
accreditation  proponent  for  one  or  more  of  the  models  requiring 
accreditation  will  prepare  a  list  of  such  models  (with  associated 
types  of  applications;  and  associated  model  versions,  if 
appropriate)  and  submit  it  to  MISMA.  Enclosure  3  offers  an 
initial  list  for  such  purposes;  however,  that  list  requires 
careful  review  from  the  following  perspectives:  what  organization 
is  the  proper  V&V  proponent  for  each  model,  which  models  require 
accreditation  (additions  or  deletions  may  be  appropriate) ,  and 
what  organization  is  the  proper  accreditation  proponent  for  each 
model.  Within  the  subsequent  three  months,  each  such  agency  will 
develop  a  one-time-only,  time-phased  plan  (including  acceptability 
criteria)  for  initial  accreditation  of  those  models,  and  submit 
the  plan  through  MISMA  for  approval  by  the  Army  Models  Committee. 
(It  is  recognized  that  use  of  a  model  may  be  necessary  before 
initial  accreditation  activities  are  completed.) 

(2)  Future  model.  Subsequent  to  distribution  of  this 
memorandum,  if  a  newly  developed  or  existing  model  is  proposed  for 
first  use  in  one  of  the  types  of  applications  listed  in  paragraph 
5h,  then  an  accreditation  proponent  will  be  identified  (in 
accordance  with  Enclosure  2,  paragraph  11),  and  that  proponent 
will  prepare  and  forward  an  accreditation  plan  (including 
acceptability  criteria)  through  MISMA  for  approval  by  the  Army 
Models  Committee. 

d.  Subsequent  accreditation.  A  model  that  has  been  judged 
accredited  for  a  particular  type  of  application  is  subject  to  re¬ 
accreditation  when  it  is  proposed  for  a  new  type  of  application, 
when  a  new  reference  version  is  released,  or  when  the  Army  Models 
Committee  determines  that  a  sufficient  period  has  passed  to 
necessitate  accreditation  activities  to  be  performed  again.  As 
always,  accreditation  will  be  preceded  by  preparation  of  a  plan 
and  forwarding  it  through  MISMA  for  approval  by  the  Army  Models 
Committee.  A  modex  that  has  been  judged  non-accredited  (i.e., 
unacceptable)  for  a  particular  type  of  application  is  subject  to 
accreditation  as  soon  as  appropriate  changes  have  been  made  to  the 
model. 

7.  RESPONSIBILITIES. 

a.  ViV  responsibility.  The  Army  agency  responsible  for 
performing  V&V  for  a  particular  Army  model  is  the  "V&V  proponent" 
as  defined  in  Enclosure  2.  The  V4V  proponent  may  seek  the 
assistance  of  another  agency;  e.g.,  the  developer  if  a  different 
agency,  or  agencies  with  functional  area  expertise.  If  the  V6V 
proponent  is  the  same  as  the  model  developer  (or  is  the  single 
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model  user),  then  in  order  to  foster  objectivity,  independent 
input  to  the  V£V  process  should  be  sought  from  other  elements  of 
the  agency  or  from  other  agencies.  VSV  proponents  for  selected 
models  are  suggested  in  Enclosure  3. 


b.  Accreditation  responsibility.  The  Army  agency  responsible 
for  accreditation  of  a  model  is  the  "accreditation  proponent"  as 
defined  in  Enclosure  2.  The  accreditation  proponent  prepares  an 
accreditation  plan  (including  acceptability  criteria)  and  is  the 
approval  authority  for  accreditation.  The  accreditation  proponent 
chooses  the  agency  that  is  to  provide  the  technical  and 
administrative  justification  of  the  accreditation  decision. 
Frequently,  the  agency  providing  that  justification  will  be  the 
same  as  the  V4V  proponent.  The  accreditation  proponent  will  be 
different  from  the  developers  and  users  of  the  model,  and 
generally  at  a  management  level  higher  than  the  V4V  proponent . 
Accreditation  proponents  for  selected  models  are  suggested  in 
Enclosure  3 . 

c.  Guidance  and  approval.  The  Army  Models  Committee  will 
provide  guidance  on  V4V  and  accreditation.  The  Committee  will 
approve  accreditation  plans  (paragraph  6c  and  6d) .  Annually,  the 
Committee  will  review  the  list  of  models  requiring  accreditation 
and  their  accreditation  status,  and  ma)ce  recommendations  for 
additional  accreditation  activities.  If  a  model  requiring 
accreditation  has  been  judged  non-accredited  (i.e.,  unacceptable) 
for  a  particular  type  of  application  and  is  proposed  for  use  for 
such  an  application,  then  a  written  request  must  be  forwarded  to, 
and  written  approval  obtained  from,  the  chairperson  of  the 
Committee . 


d.  Monitoring.  The  MISMA  will  coordinate  accreditation  plans 
(see  paragraph  6c  and  6d) .  Annually,  the  MISMA  will  publish  a 
list  of  models  requiring  accreditation  and  their  accreditation 
statuses  (with  associated  types  of  applications) . 


Uente 

Technical  Advisor  to  the 
Deputy  Chief  of  Staff  for 
Operations  and  Plans 


Walter  W.  Hollis 

Deputy  Under  Secretary  of  the  Army 
(Operations  Research) 


Ends 
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DISTRIBUTION: 

HQDA 

SAFM-ZA 

SAIL 

SAIS-ZA 

SARD-ZA 

SARD- IP 

DACS-DPZ-A 

DALO-ZA 

DALO-PLA 

DAMI-ZA 

DAMI-FIT 

DAMO-ZA 

D AMO- ZD 

DAMO-TRS 

DAPE-ZA 

DASG-ZA 

Commander 

U.  S.  Army  Materiel  Command 

U.  S.  Army  Laboratory  Command 

U.  S.  Army  Missile  Command 

U.  S.  Army  Tank  Automotive  Command 

U.  S.  Army  Test  and  Evaluation  Command 

U.  S.  Army  Training  and  Doctrine  Command 

U.  S.  Army  Logistics  Center 

U.  S.  Army  Soldier  Support  Center 

U.  S.  Army  Combined  Arms  Center 

U.  S.  Army  Combined  Arms  Combat  Developments  Activity 
U.  S.  Army  Combined  Arms  Training  Activity 
and  National  Simulation  Center 
TRADOC  Analysis  Command 
TRADOC  Test  and  Experimentation  Command 
Army  Training  Support  Center 
U.  S.  Army  Intelligence  Agency 
U.  S.  Total  Army  Personnel  Command 
U.  S,  Army  Operational  Test  and  Evaluation  Agency 
U.  S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences 
Engineer  Study  Center 
U.  S.  Logistics  Evaluation  Agency 
Joint  Warfare  Center 
Warrior  Preparation  Center 
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DISTRIBUTION  (continued) : 

Director 

U.  S.  Army  Concepts  Analysis  Agency 

U.  S.  Army  Materiel  Systems  Analysis  Activity 

U.  S.  Army  Ballistics  Research  Laboratory 

U.  S.  Army  Model  Improvement  and  Study  Management  Agency 

U.  S.  Army  Force  Integration  Support  Agency 

Joint  Technical  Coordinating  Group/SS 

Institute  for  Defense  Analyses 

Jet  Propulsion  Laboratory 

Lawrence  Livermore  National  Laboratory 

Los  Alamos  National  Laboratory 

Oak  Ridge  National  Laboratory 

MITRE  Corporation 

RAND  Arroyo  Center 

Commandant 

U.  S.  Army  Air  Defense  Artillery  Center  and  School 

U.  S.  Army  Armor  Center  and  School 

U.  S.  Army  Command  and  General  Staff  College 

U.  S.  Army  Field  Artillery  School 

U.  S.  Airmy  Logistics  Management  College 

U.  S.  Army  War  College 

Project  Manager,  Training  Devices 

Organization  of  the  Joint  Chiefs  of  Staff 

Director,  Operational  Plans  and  Interoperability 
Directorate  (J-7) 

Director,  Force  Structure,  Resources,  and  Assessment 
Directorate  (J--8) 
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DEFINITIONS 


The  following  definitions  apply  to  the  main  body  of  the 
memorandum : 


1.  Model .  An  abstract  representation  of  a  real-world  entity 
(object,  system,  activity,  or  situation),  including,  as 
appropriate,  subsystems  and  their  interrelationships.  For  this 
discussion,  a  model  is  further  restricted  to  one  that  is 
implemented  as  a  computer  program. 

2.  Simulation.  A  model  that  represents  activities  and 
interactions  over  time.  A  simulation  may  be  fully-automated 
(i.e.,  it  executes  without  human  intervention),  or  it  may  be 
interactive  or  interruptible  (i.e.,  the  user  may  intervene  during 
execution) . 

3.  Type  of  model  characterized  by  frequency  of  use. 

a.  Ad  hoc  model.  A  model  that  is  developed  to  address  a 
single,  specific  issue. 

b.  Standing  model.  A  model  that  is  developed  to  address  a 
generic  class  of  problems  and  is  maintained  on  a  continual  basis 
to  support  multiple  or  recurring  applications. 

4.  Verification .  The  process  of  determining  that  a  model 
accurately  represents  the  developer's  conceptual  description  and 
specifications.  In  a  large-scale  model  development,  verification 
is  applied  at  each  stage  to  ensure  that  the  products  of  that  stage 
accurately  implement  the  specifications  from  the  previous  stage. 

5.  Validation .  The  process  of  determining  that  a  model  is  an 
accurate  representation  of  the  intended  real-world  entity  from  the 
perspective  of  the  intended  use  of  the  model . 

6.  Configuration  management.  A  discipline  applying  technical  and 
administrative  oversight  and  control  to  identify  and  document  the 
functional  requirements  and  capabilities  of  a  model,  control 
changes  to  those  capabilities,  and  document  and  report  the 
changes.  Configuration  management  includes  ensuring  that  the 
detailed  design  and  the  computer  source  code  of  the  model  are 
properly  documented.  (This  definition  and  the  next  one  have  been 
adapted  from  AR  70-37,  reference  15) . 

7.  Configuration  control.  That  element  of  configuration 
management  that  involves  the  systematic  evaluation,  coordination, 
approval  or  disapproval,  and  implementation  of  changes  to  the 
model  source  code.  Included  activities  are — 

a.  Maintaining  the  reference  version  of  the  source  code. 
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b.  Controlling  changes  to  the  reference  version  and 
dissemination  of  those  changes. 

c.  Ensuring  that  documentation  of  the  model  is  up-to-date. 

8.  Accreditation .  Certification  that  a  model  is  acceptable  for 
use  for  a  specific  type(s)  of  application (s) .  Accreditation  is 
approval  by  management — based  on  experience  and  expert  judgement — 
that  a  model  is  adequate  for  its  intended  use.  The  accreditation 
mechanism  recognizes  that  V&V  of  a  model  are  continual  processes 
and  that  full  validation  of  the  model  may  not  be  technically  or 
economically  feasible.  Hence,  accreditation  is  possible  even  if 
the  model  is  not  fully  validated.  However,  accreditation  does  not 
lessen  the  need  for  continuing  to  work  toward  full  V&V. 

9.  V&V  documentation.  Documentation  of  V&V  activities  includes  a 
written  assessment  plan  and  a  written  report . 

a.  Plan.  The  assessment  plan  has  two  components. 

(1)  Management  plan.  This  component  describes  the  tasks, 
schedule,  and  (personnel  and  computer)  resource  requirements. 

(2)  Analysis  plan.  This  component  describes  scope, 
limitations,  constraints,  techniques  to  be  employed,  types  and 
sources  of  data  to  be  collected,  and  acceptability  criteria. 

b.  Report .  The  report  documents  V&V  activities  and  includes 
the  following;  brief  descriptions  of  the  real-world  entity  to  be 
represented,  intended  purpose  of  the  model,  and  design 
requirements;  description  of  V&V  activities  performed;  extent  to 
which  model  meets  acceptability  criteria;  and  recommendations. 

iir.  V&V  proponent .  The  Army  agency  responsible  for  performing  V&V 
on  a  given  model  (and  version  of  the  model,  if  there  are  multiple 
versions) .  The  V&V  proponent  may  seek  the  assistance  of  another 
agency;  e.g.,  the  developer  if  a  different  agency,  or  agencies 
with  functional  area  expertise.  Which  agency  is  the  V&V  proponent 
depends  on  whether  the  model  is  in  development  or  has  been 
previously  developed,  who  is  developing  (or  developed)  the  model, 
who  is  using  it,  and  who  is  maintaining  it.  The  following  cases 
apply: 

a.  Model  that  is  under  Army  control  and  for  which  the  Army  V&V 
proponent  has  been  established  previously.  That  agency  is 
responsible  for  V&V.  Examples  are  the  members  of  FAMSIM. 

b.  Model  for  which  the  Army  V&V  proponent  has  not  been 
established,  and  that  is  being  developed  in-house  by,  or  that  is 
being  developed  under  a  cont  act  sponsored  by,  an  Army  agency. 

The  developing  (or  sponsoring)  Army  agency  is  responsible  for  V&V. 
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c.  Previously  developed  model  that  is  under  Army  control,  for 
which  the  Army  V&V  proponent  has  not  been  established,  that  is  in 
use  by  the  Army,  and  that  has  a  single  Army  agency  as  the  user  and 
maintainer.  That  agency  is  responsible  for  V&V. 

d.  Previously  developed  model  that  is  under  Army  control,  for 
which  the  Army  V&V  proponent  has  not  been  established,  that  is  in 
use  by  the  Army,  and  for  which  the  Army  user  agency  is  different 
from  the  maintaining  agency  or  for  which  there  are  multiple  Army 
user  agencies.  The  configuration  control  (users')  group,  if  it 
exists  and  is  under  Army  control,  is  responsible  for  V&V. 

Otherwise,  the  primary  Army  user  is  responsible. 

11.  Accreditation  proponent.  The  Army  agency  responsible  for 
accreditation;  in  particular,  for  setting  acceptability  criteria 
and  being  the  approval  authority.  The  cases  below  determine  which 
agency  has  this  responsibility. 

a.  Model  that  is  or  will  be  used  by  the  Army.  The  Army  agency 
that  is  or  will  be  the  primary  customer  of  the  analysis  products 

of  the  model  has  accreditation  responsibility.  For  a  model 
developed  and  used  by  an  FFRDC  or  contracting  firm  in  a  major 
study  sponsored  by  the  Army,  the  Army  sponsor  is  the  accreditation 
proponent . 

b.  Model  that  affects  the  Army,  that  is  not  under  Army 
control,  and  that  is  not  used  by  the  Army.  The  Army  agency  that 
is  the  proponent  for  the  appropriate  mission  area,  function,  or 
activity  has  accreditation  responsibility. 
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MODELS  SUITABLE  FOR  V&V  AND  ACCREDITATION 


Below  is  a  list  of  models  suitable  for  ViV  and  accreditation. 
Several  notes  and  caveats  apply — 

a.  The  models  are  only  examples;  i.e.,  the  list  is  not  all- 
inclusive.  Furthermore,  the  designation  of  V&V  proponent, 
identification  as  accreditation  candidate,  and  designation  of 
accreditation  proponent,  are  subject  to  review  and  update.  In 
particular,  where  a  model  is  not  identified  as  an  accreditation 
candidate,  the  implication  is  that  the  model  does  not  fall  under  a 
category  of  paragraph  5h  of  the  memorandum  and  hence  is  not 
subject  to  accreditation  (at  least  for  the  memorandum).  If  the 
model  is  in  fact  in  one  of  the  categories  of  paragraph  5h,  then  an 
accreditation  proponent  must  be  identified  in  accordance  with 
Enclosure  2,  paragraph  11. 

b.  Examples  are  not  repeated.  For  instance,  JANUS (T)  is  used 
for  major  Army  COEAs,  and  thus  is  shown  under  first  category. 

JANUS (T)  is  also  used  for  training,  testing,  and  education,  but  is 
not  shown  under  those  categories . 

c.  Where  a  V&V  proponent  is  not  designated,  the  Army  does  not 
control  the  model  and  hence  cannot  designate  what  organization  is 
to  perform  V&V.  In  such  a  case,  however,  an  Army  accreditation 
proponent  is  identified.  Where  the  model  is  developed  and  used  by 
an  FFRDC  or  contracting  firm  in  support  of  a  major  Army  study,  the 
Army  sponsor  is  the  accreditation  proponent. 


CATEGORY 

VERIFICATION 

AND  VALIDATION 

ACCREDITATION 

CANDIDATE 

CANDIDATE 

Model  used 

ADAM 

Yes 

CAC 

for  major 

BASEWAM 

• 

Yes 

CAC 

Army  COEA 

CARMO 

TRAC 

Yes 

CAC 

CASTFOREM 

TRAC 

Yes 

CAC 

CISCIAO 

USAADASCH 

Yes 

CAC 

COMO 

USAADASCH 

Yes 

CAC 

CORBAN 

TRAC 

Yes 

CAC 

DC2M 

Yes 

CAC 

EAGLE 

TRAC 

Yes 

CAC 

ELAN 

TRAC 

Yes 

CAC 

JANUS (T) 

TRAC 

Yes 

CAC 

LDM 

USALOGC 

Yes 

ODCSLOG 

TAFSIM 

USAFAS 

Yes 

CAC 

TANK  WARS 

USAARMC 

Yes 

CAC 

VIC 

TRAC 

Yes 

CAC 
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CATEGORY 

VERIFICATION 

AND  VALIDATION 

ACCREDITATION 

QF  MODEL 

CANDIDATE 

PROPONENT 

CAWDIPATE 

PROPONENT 

Model  used 

AFP 

CAA 

Yes 

ODCSOPS 

for  major 

CEM 

CAA 

Yes 

ODCSOPS 

4 

Army  PPBES 

FASTALS 

CAA 

Yes 

ODCSOPS 

study 

FORCEM 

CAA 

Yes 

ODCSOPS 

WRAP 

AMSAA 

Yes 

HQ  AMC 

t 

* 

Model 

TACSAGE 

Yes 

CAA 

developed 

TACWAR 

Yes 

USAWC 

P 

and  used  by 

RSAS 

Yes 

USAWC 

FFRDC  or 

contractor 

Training 

ARTBASS 

TRAC 

Yes 

CATA 

simulation 

ASSAULT 

USALOGC 

Yes 

CATA 

BBS 

TRAC 

Yes 

CATA 

CCTT 

TRAC 

Yes 

CATA 

CSSTSS 

TRAC 

Yes 

CATA 

DBIT 

TRAC 

Yes 

CATA 

FB:B-C 

TRAC 

Yes 

CATA 

TACSIM 

TRAC 

Yes 

CATA 

Joint  model 

JESS 

TRAC 

Yes 

CATA 

JTLS 

Yes 

USAWC 

Joint  Warfare 

System 

Yes 

SOTACA 

Yes 

USAWC 

TAM 

Yes 

Model  used 

ADATS  (6 -DOF) 

MICOM 

Yes 

HQ  AMC 

for  testing 

ALWSIM  III 

LABCOM 

AMM 

TACOM 

Yes 

HQ  AMC 

ARTQUIK 

JTCG/SS 

AURA 

BRL 

Yes 

LABCOM 

DBNUSSE 

BRL 

Yes 

LABCOM 

EIEM 

TECOM 

EVADE  II 

AMSAA 

GAMES  I 

AMSAA 

Yes 

HQ  AMC 

GIFT 

BRL 

Yes 

LABCOM 

HEAT 

ARI 

HELMATES  II 

AMSAA 

Yes 

HQ  AMC 

HWXL 

MICOM 

Yes 

HQ  AMC 

lEW/FAM 

TRAC 

Yes 

CAC 

MOSES  II 

AMSAA 

MSE  PAM.;; 

EPG 

Yes 

TECOM 

PARACOMPT 

BRL 

Yes 

LABCOM 

REGSIM 

AMSAA 

Yes 

HQ  AMC 

RMP  (6-DOF) 

MICOM 

Yes 

HQ  AMC 

SAMSMAE 

AMSAA 

Yes 

HQ  AMC 

SAMSITE 

AMSAA 

Yes 

HQ  AMC 

SESAME 

AMSAA 

Yes 

ODCSLOG 

TARMS  II 

CAA 

Yes 

ODCSLOG 

VAST 

BRL 

Yes 

LABCOM 

♦ 
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CATEGORY 

VERIFICATION  AND 

VALIDATION 

ACCREDITATION 

QE  MODEL 

CANDIDATE 

PROPONENT 

CAMPIDATE 

PROPONENT 

Decision 

ALBM 

CACDA 

support 

BDEPLANNER 

TRAC 

system 

Force  Builder 

USAFISA 

Yes 

ODCSOPS 

Organize  the  War 

CAA 

Yes 

ODCSOPS 

SABRE 

PAED 

TAEDP 

LEA 

Yes 

ODCSLOG 

TOPSIS 

PERSCOM 

Yes 

PAED 

TBD  TBD 
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V&V  PROCEDURES 


Useful  V4V  procedures  are  shown  here,  categorized  by  the  life 
cycle  stage  of  the  model.  For  the  purposes  of  V&V,  the  life  cycle 
of  a  model  is  described  here  in  terms  of  the  stages  identified  in 
AMC-P  702-xx  (reference  18) .  An  alternate  decomposition  of  the 
life  cycle,  applicable  to  the  procurement  of  information  systems 
and  found  in  AR  25-3  (reference  17),  is  not  used  here. 

1.  Life  cycle  stage.  Model  concffpt . 

Basic  description.  The  entity  to  be  represented,  the  purpose 
of  the  model,  and  the  general  approach  to  representing  the  entity 
are  described. 

V&V  procedures.  Develop  an  understanding  of  the  real-world 
entity  to  be  represented.  Assess  the  adequacy  of  the  planned 
representation . 

2.  Life  cycle  stage.  Requirements  definition. 

Basic  description.  Functional,  operational,  interface,  and 
performance  requirements  are  defined. 

V&V  procedures.  Assess  the  extent  to  which  requirements  match 
the  intended  real-world  entity.  Initiate  the  written  V&V  plan. 
Establish  acceptability  criteria, 

3.  Life  cycle  stage.  Preliminary  design. 

Basic  description.  The  overall  architecture  of  the  model  is 
defined.  Technical  approaches  are  described. 

V&V  procedures.  Review  tlje  architecture  and  theoretical 
foundations  to  assess  the  adequacy  of  the  preliminary  design 
against  requirements.  Review  the  types  and  formats  of  required 
input  data  for  appropriateness  and  availability.  Update  the  V&V 
plan.  Initiate  the  written  V&V  report. 

4.  Life  cycle  stage.  Detailed  design. 

Basic  description.  The  architectural  design  is  refined. 
Embedded  algorithms  are  described. 

V&V  procedures.  '  Review  the  architecture,  theoretical 
approaches,  embedded  algorithms,  and  underlying  assumptions  to 
assess  the  adequacy  of  the  detailed  design  against  the  preliminary 
design.  Review  the  representation  of  threat  doctrine  and  tactics. 
Update  the  V&V  plan  and  report . 
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5.  Life  cycle  stage. 


Coding • 


Basic  description.  The  model  is  converted  to  computer  source 
code. 

% 

V4V  procedures.  Review  the  source  code  to  assess  the  adequacy 
of  the  code  against  the  detailed  design.  Included  here  are  i 

techniques  such  as  a  structured  walk-through,  line-by-line  code 
review,  and  inclusion  of  computer-generated  warning  messages  and 
review  of  the  results  of  such  messages.  Perform  test  runs  (of 
components  and  of  the  complete  model)  to  compare  the  results  of 
the  model  with  those  known  of  (or  anticipated  of)  the  real-world 
entity.  Update  the  V&V  plan  and  report. 

6.  Life  cycle  stage.  Testing. 

Basic  description.  The  model  is  tested  for  acceptability. 

V&V  procedures .  Review  input  data  for  accuracy  and 
completeness.  Perform  test  runs,  and  subsequently  examine  program 
logic  flow,  data  values  internal  to  the  model,  output  data 
generated  by  the  model,  results  of  computer-generated  warning 
messages,  and  graphical  displays  of  computer  output.  Determine 
the  sensitivity  of  output  data  to  input  data.  Compare  output  data 
with  real-world  data  (including  historical  data) ;  output  data  from 
validated  models;  test,  exercise,  and  training  data;  and 
operations  data.  (Other  models  and  data  from  tests  and  exercises 
are  most  useful  for  "calibrating"  the  given  model;  i.e.,  making 
adjustments  to  input  data  or  model  logic  to  obtain  closer 
agreement  with  an  external  index.)  Update  the  V&V  plan  and 
report . 

7.  Life  cycle  stage.  Application  and  post-deployment  software 
support  (PDSS) . 

• 

Basic  description.  The  model  is  applied,  and  is  updated  to 
account  for  changes  in  the  real-world  entity  being  represented  or 
to  incorporate  representations  of  additional  entities. 

V&V  procedures.  Perform  additional  V&V  activities. 

Appropriate  procedures  include  those  listed  under  testing  plus 
peer  review jby  an  outside  panel  of  experts.  Update  the  V&V  plan 
and  report . 
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